A Platform for Automating Chaos Experiments

The Netflix video streaming system is composed of many interacting services. In such a large system, failures in individual services are not uncommon. This paper describes the Chaos Automation Platform, a system for running failure injection experiments on the production system to verify that failures in non-critical services do not result in system outages.Comment: Conference publicatio

Development of An Empirical Approach to Building Domain-Specific Knowledge Applied to High-End Computing

This dissertation presents an empirical approach for building and storing knowledge about software engineering through human-subject research. It is based on running empirical studies in stages, where previously held hypotheses are supported or refuted in different contexts, and new hypotheses are generated. The approach is both mixed-methods based and opportunistic, and focuses on identifying a diverse set of potential sources for running studies. The output produced is an experience base which contains a set of these hypotheses, the empirical evidence which generated them, and the implications for practitioners and researchers. This experience base is contained in a software system which can be navigated by stakeholders to trace the "chain of evidence" of hypotheses as they evolve over time and across studies. This approach has been applied to the domain of high-end computing, to build knowledge related to programmer productivity. The methods include controlled experiments and quasi-experiments, case studies, observational studies, interviews, surveys, and focus groups. The results of these studies have been stored in a proof-of-concept system that implements the experience base

A Pilot Study to Evaluate Development Effort for High Performance Computing

The ability to write programs that execute efficiently on modern parallel computers has not been fully studied. In a DARPA-sponsored project, we are looking at measuring the development time for programs written for high performance computers (HPC). To attack this relatively novel measurement problem, our goal is to initially measure such development time in student programming to evaluate our own experimental protocols. Based on these results, we will generate a set of feasible experimental methods that can then be applied with more confidence to professional expert programmers. This paper describes a first pilot study addressing those goals. We ran an observational study with 15 students in a graduate level High Performance Computing class at the University of Maryland. We collected data concerning development effort, developer activities and chronology, and resulting code performance, for two programming assignments using different HPC development approaches. While we did not find strong correlations between the expected factors, the primary outputs of this study are a set of experimental lessons learned and 12 wellformed hypotheses that will guard future study

An Environment of Conducting Families of Software Engineering Experiments

The classroom is a valuable resource for conducting software engineering experiments. However, coordinating a family of experiments in classroom environments presents a number of challenges to researchers. This paper describes an environment that simplifies the process of collecting, managing and sanitizing data from classroom experiments, while minimizing disruption to natural subject behavior. We have successfully used this environment to study the impact of parallel programming languages on programmer productivity at multiple universities across the United States

Position Paper and Brief Announcement: An Empirical Study to Compare Two Parallel Programming Models

While there are strong beliefs within the community about whether one particular parallel programming model is easier to use than another, there has been little research to analyze these claims empirically. Currently, the most popular paradigm is message-passing, as implemented by the MPI library [1]. However, MPI is considered to be difficult for developing programs, because it forces the programmer to work at a very low level of abstraction. One alternative parallel programming model is the PRAM model, which supports fine-grained parallelism and has a substantial history of algorithmic theory [2]. It is not possible to program current parallel machines using the PRAM model because modern architectures are not designed to support such a model efficiently. However, current trends towards multicor